Many of our test harnesses are Python 3 compatible, though they still run with Python 2 in CI since mozharness hasn't been converted yet.

We should set things up such that tasks can specify which version of Python they should run with (similar to the source-test kind). This way we'll be able to easily identify what failures in mozharness need to be addressed in order to turn them on in CI with Python 3.

This will also allow us to convert the test portions of bug 1532427 piece by piece, rather than all at once.

Backed out for perm failures.

Push with failure: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=LELFEimARoqbIM3y0d1elA.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=a39c15378e6f46803c949bc076cf796dfef3c02f&searchStr=mochitest&failure_classification_id=2

Log: https://treeherder.mozilla.org/logviewer?job_id=322817150&repo=autoland&lineNumber=2543

Backout: https://hg.mozilla.org/integration/autoland/rev/cd741763d086c5988a926b2bcc85f1d362d9442d

I confirmed again that the only diff in the taskgraph was an additional PYTHON=python2.7 environment variable. As far as I can tell, this env should only be used in test-linux.sh where the value it is replacing was also python2.7. I have no idea how this patch could have caused an intermittent failure anywhere :/..

I guess I'll try renaming the environment variable to make sure there's nothing else that was using it.

The errors are still reproducible after changing the environment:
https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=OLbN-HxLTg-xpWmYUPMDtw.0&revision=341c26337ed808b888b48f838074847cf06e4c58

There's something fishy going on here.. I'm 99.2% sure that these failures aren't the result of my patch, but rather something to do with docker-worker. I filed bug 1679467 to investigate further.

This same error was caused by a patch that only touches a comment in test-linux.sh. So once bug 1679467 is fixed, this can be re-landed.

https://hg.mozilla.org/mozilla-central/rev/f5532b7dc480

Backed out as requested: https://hg.mozilla.org/integration/autoland/rev/6bac7ddc9a73ce0ba033093fc1838bb315d5655c

The backout was for bug 1679952.

After this change, Android 4.0 API16+ pgo Bpgo(run) tasks https://treeherder.mozilla.org/jobs?repo=autoland&tochange=a682e468c5032c67086679b995003ef9285aa92d&fromchange=70858e3aab9c04e59234f51cde2a9d9a9dea9902&searchStr=run&selectedTaskRun=Kb0MxCLFQBGJzr0z6kq0HQ.0

have 2 exceptions like this:

[task 2020-11-30T19:43:02.329Z] 19:43:02     INFO - run-tests - Wall time: 892s; CPU: 25%; Read bytes: 262144; Write bytes: 1362161664; Read time: 0; Write time: 60016
[task 2020-11-30T19:48:02.347Z] Exception ignored in: <bound method Marionette.__del__ of <marionette_driver.marionette.Marionette object at 0x7fae1816f5f8>>
[task 2020-11-30T19:48:02.347Z] Traceback (most recent call last):
[task 2020-11-30T19:48:02.347Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/marionette.py", line 532, in __del__
[task 2020-11-30T19:48:02.347Z]     self.cleanup()
[task 2020-11-30T19:48:02.347Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/marionette.py", line 517, in cleanup
[task 2020-11-30T19:48:02.347Z]     self.delete_session()
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/marionette.py", line 1220, in delete_session
[task 2020-11-30T19:48:02.348Z]     self._send_message("WebDriver:DeleteSession")
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/decorators.py", line 37, in _
[task 2020-11-30T19:48:02.348Z]     m._handle_socket_failure()
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/marionette.py", line 684, in _handle_socket_failure
[task 2020-11-30T19:48:02.348Z]     returncode = self.instance.runner.wait(timeout=self.shutdown_timeout)
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/device.py", line 153, in wait
[task 2020-11-30T19:48:02.348Z]     while self.is_running():
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/runner.py", line 186, in is_running
[task 2020-11-30T19:48:02.348Z]     return self.returncode is None
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/device.py", line 136, in returncode
[task 2020-11-30T19:48:02.348Z]     if self.app_ctx.device.process_exist(self.app_ctx.remote_process):
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/adb.py", line 3118, in process_exist
[task 2020-11-30T19:48:02.349Z]     if self._pidof(app, timeout=timeout):
[task 2020-11-30T19:48:02.349Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/adb.py", line 1133, in _pidof
[task 2020-11-30T19:48:02.349Z]     procs = self.get_process_list(timeout=timeout)
[task 2020-11-30T19:48:02.350Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/adb.py", line 2951, in get_process_list
[task 2020-11-30T19:48:02.350Z]     raise ADBTimeoutError("%s" % adb_process)
[task 2020-11-30T19:48:02.350Z] mozdevice.adb.ADBTimeoutError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb wait-for-device shell ps; echo adb_returncode=$?, exitcode: None, stdout: 

and the Bpgo(run) task times out frequently on Android.

Backout merged: https://hg.mozilla.org/mozilla-central/rev/6bac7ddc9a73

bc: are you looking at this? It looks like maybe the short term solution is to ensure we set PYTHON=python2.7 before invoking the tests. The actual error seems like it could be a change in the way the gc runs under py3? Having stuff in del methods is pretty scary... Hopefully there's some more straightforward solution :)

I hadn't planned on it though I had hoped to make use of this in my other python3 work. I haven't looked at the patch closely but it seems weird this is showing up as an ADBTimeoutError on Android. Is this the only reason the patch was backed out?

I'll keep the NI active to remind me and will revisit after I get my other python3 work landed.

This is I think the main thing blocking using wpt as Py3 which is on a bit of a deadline. So I can look and see if there's a simple workaround here.

jgraham: I think your approach in bug 1680752 is the approach to take where we run under python2.7 by default unless we've explicitly requested the test run under python3. There are many places in the tree even after my recent work on python3 compatibility where the various packages do not yet support python3. Attempting to switch to python3 before we resolve those issues will only lead to pain.

Looks like bug 1678663 implemented a way to specify Python 3 in the configs in the meantime here. While it's opt-in rather than opt-out, it serves the same purpose and is likely good enough.